منابع مشابه
Building scalable digital library ingestion pipelines using microservices
CORE, a harvesting service offering access to millions of open access research papers from around the world, has shifted its harvesting process from following a monolithic approach to the adoption of a microservices infrastructure. In this paper, we explain how we rearranged and re-scheduled our old ingestion pipeline, present CORE’s move to managing microservices and outline the tools we use i...
متن کاملBuilding reliable and efficient data transfer and processing pipelines
Scientific distributed applications have an increasing need to process and move large amounts of data across wide area networks. Existing systems either closely couple computation and data movement, or they require substantial human involvement during the end-to-end process. We propose a framework that enables scientists to build reliable and efficient data transfer and processing pipelines. Ou...
متن کاملLet It Flow: Building Mashups with Data Processing Pipelines
Mashups are a new kind of interactive Web application, built out of the composition of two or more existing Web service APIs and data sources. Whereas “pure” mashups are built relying entirely on the Web browser as a platform for integration at the presentation layer, this approach is not always feasible and part of the mashup integration logic must be deployed on the Web server instead. In the...
متن کاملBuilding Highly-Optimized, Low-Latency Pipelines for Genomic Data Analysis
Next-generation sequencing has transformed genomics into a new paradigm of data-intensive computing. The deluge of genomic data needs to undergo deep analysis to mine biological information. Deep analysis pipelines often take days to run, which entails a long cycle for algorithm and method development and hinders future application for clinic use. In this project, we aim to bring big data techn...
متن کاملBuilding Global Knowledge Pipelines: The Role of Temporary Clusters
Business people and professionals come together regularly at trade fairs, exhibitions, conventions, congresses, and conferences. Here, their latest and most advanced findings, inventions and products are on display to be evaluated by customers and suppliers, as well as by peers and competitors. Participation in events like these helps firms to identify the current market frontier, take stock of...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Nature
سال: 2001
ISSN: 0028-0836,1476-4687
DOI: 10.1038/35092705